Technical performance data trained learning system

ABSTRACT

Various embodiments are described that relate to an adaptive learning system. The adaptive learning system can be trained by correlation between a first set of raw technical performance data and a set of actual operational effectiveness assessment data. Once trained, the adaptive learning system can be deployed. Once deployed, the adaptive learning system can produce a set of predicted operational effectiveness assessment data from a second set of raw technical performance data that is different from the first set of raw technical performance data.

GOVERNMENT INTEREST

The innovation described herein may be manufactured, used, imported,sold, and licensed by or for the Government of the United States ofAmerica without the payment of any royalty thereon or therefor.

BACKGROUND

A vast amount of information can be made available to decision makers.The value of this information can be based, at least in part, on theability of those decision makers to understand that information and makedeterminations based on the understanding of the information. If thedecision makers misunderstand the information, then incorrect decisionscan be made. Depending on the importance of these decisions lives can beat risk.

SUMMARY

In one embodiment, a method comprises obtaining a set of raw technicalperformance data. The method also comprises predicting, by way of alearning system, a set of predicted operational assessment data based,at least in part, on the set of raw technical performance data. Thelearning system can be trained from a set of training technicalperformance data and a set of training operational assessment data.

In one embodiment, a computer-implemented method comprises accessing aset of raw technical performance data. In addition, thecomputer-implemented method comprises accessing a set of actualoperational effectiveness data. Further, the computer-implemented methodcomprises training an adaptive learning system based on a relationshipbetween at least part of the set of raw technical performance data andthe set of actual operational effectiveness data.

In one embodiment, a method comprises predicting a first operationalscore for a first individual service by way of a first service adaptivelearning system. The method also comprises predicting a secondoperational score for a second individual service by way of a secondservice adaptive learning system that is distinct from the first serviceadaptive learning system. Additionally, the method comprises predictingan overall operational effectiveness though use of an operationaladaptive learning system, where the overall operational effectiveness isbased, at least in part, on the first operational score and the secondoperational score.

BRIEF DESCRIPTION OF THE DRAWINGS

Incorporated herein are drawings that constitute a part of thespecification and illustrate embodiments of the detailed description.The detailed description will now be described further with reference tothe accompanying drawings as follows:

FIG. 1 illustrates one embodiment of an environment comprising anadaptive learning system subjected to training;

FIG. 2 illustrates one embodiment of an environment comprising theadaptive learning system in its deployed phase of operations;

FIG. 3 illustrates one embodiment of an environment comprising theadaptive learning system during both training and deployment phases ofoperation;

FIG. 4 illustrates one embodiment of a configuration comprising a numberof systems and a parent system for system of systems (SoS) operationaleffectiveness prediction;

FIG. 5 illustrates one embodiment of an environment of SoS operationaleffectiveness predicting 2-tiered learning system architecture;

FIG. 6 illustrates one embodiment of an environment in practicalapplication of predicting operational effectiveness of platform positionSoS;

FIG. 7 illustrates one embodiment of a system comprising a processor anda non-transitory computer-readable medium;

FIG. 8 illustrates one embodiment of a method comprising three actions;

FIG. 9 illustrates one embodiment of a method comprising six actions;and

FIG. 10 illustrates one embodiment of a method comprising two actions.

DETAILED DESCRIPTION

Before a major decision is made it can be beneficial to predict theoutcome of the decision and/or predict operational effectiveness of asystem associated with the decision. Aspects disclosed herein can beused in making a successful prediction of operational performancefactors of complex systems. In order to make a successful prediction atrained learning system can be employed. The learning system canunderstand a relationship between technical performance data (e.g.,instrument provided information) and operational effectivenessassessment data (e.g., calculated information).

In one example, a manufacturing process can be used to create bottledsoda with a desired taste. Taste testers can be employed to provide theoperational effectiveness assessment data that describe the taste ofsoda with slightly different mixture percentages that are technicalperformance data. The following hypothetical data table could beproduced:

Water to powder mixture Significant deviation of taste 89.5% water,10.5% powder Yes 89.6% water, 10.4% powder Yes 89.7% water, 10.3% powderYes 89.8% water, 10.2% powder No 89.9% water, 10.1% powder No   90%water, 10% powder No 90.1% water, 9.9% powder No 90.2% water, 9.8%powder No 90.3% water, 9.7% powder Yes 90.4% water, 9.6% powder Yes90.5% water, 9.5% powder YesBased on the technical performance data (water to powder mixture) andthe operational effectiveness assessment data (significant deviation oftaste) a relationship can be that when the water to powder mixture is±0.2 from 90-10 the taste deviation is not significant. The learningsystem can be trained with this relationship and then deployed. Actualwater to power mixture information can be fed to the learning system andbased on this the learning system can make a direct prediction (e.g., ifthe customers will be able to taste a difference) or an indirectprediction (e.g., if the product will be successful). This predictioncan be used. In one example if the prediction is that customers will beable to taste a difference, then a batch can be discarded while if theprediction is that the customers will not be able to taste thedifference, then the batch can be bottled, packaged, and shipped.

In addition, the trained learning system can be adaptive. Continuingwith the soda making example, as customer feedback is obtained thetrained learning system can be modified, removed, fortified, etc. In oneexample, if product complaints are made from actual customers that sodawithin the plus or minus 0.3 range (89.7-10.3 or 90.3-9.7) has tastevariation, then the learning system can change based on this newinformation. Aspects disclosed herein can be practiced in a wide varietyof areas, including but not limited to manufacturing, communications,military, business, computing, finance, etc.

The following includes definitions of selected terms employed herein.The definitions include various examples. The examples are not intendedto be limiting.

“One embodiment”, “an embodiment”, “one example”, “an example”, and soon, indicate that the embodiment(s) or example(s) can include aparticular feature, structure, characteristic, property, or element, butthat not every embodiment or example necessarily includes thatparticular feature, structure, characteristic, property or element.Furthermore, repeated use of the phrase “in one embodiment” may or maynot refer to the same embodiment.

“Computer-readable medium”, as used herein, refers to a medium thatstores signals, instructions and/or data. Examples of acomputer-readable medium include, but are not limited to, non-volatilemedia and volatile media. Non-volatile media may include, for example,optical disks, magnetic disks, and so on. Volatile media may include,for example, semiconductor memories, dynamic memory, and so on. Commonforms of a computer-readable medium may include, but are not limited to,a floppy disk, a flexible disk, a hard disk, a magnetic tape, othermagnetic medium, other optical medium, a Random Access Memory (RAM), aRead-Only Memory (ROM), a memory chip or card, a memory stick, and othermedia from which a computer, a processor or other electronic device canread. In one embodiment, the computer-readable medium is anon-transitory computer-readable medium.

“Component”, as used herein, includes but is not limited to hardware,firmware, software stored on a computer-readable medium or in executionon a machine, and/or combinations of each to perform a function(s) or anaction(s), and/or to cause a function or action from another component,method, and/or system. Component may include a software controlledmicroprocessor, a discrete component, an analog circuit, a digitalcircuit, a programmed logic device, a memory device containinginstructions, and so on. Where multiple components are described, it maybe possible to incorporate the multiple components into one physicalcomponent or conversely, where a single component is described, it maybe possible to distribute that single component between multiplecomponents. Aspects disclosed herein can be performed by at least onecomponent (e.g., a trainer component, a validation component, etc.).

“Software”, as used herein, includes but is not limited to, one or moreexecutable instructions stored on a computer-readable medium that causea computer, processor, or other electronic device to perform functions,actions and/or behave in a desired manner. The instructions may beembodied in various forms including routines, algorithms, modules,methods, threads, and/or programs including separate applications orcode from dynamically linked libraries.

FIG. 1 illustrates one embodiment of an environment 100 comprising anadaptive learning system 110. The environment can be used to train thelearning system 110. A set of raw technical performance data 120 and aset of actual operational effectiveness assessment data 130 can be usedto train the adaptive learning system 110. The set of raw technicalperformance data 120 can be hardware instrumentation data (e.g., packetperformance data), condition data (e.g., signal to noise ratios,interference levels, etc.), etc. and can be provided to the adaptivelearning system 110.

In one example, a soldier can carry a backpack with a global positioningsystem (GPS). This backpack can include a data recorder that records alocation of the soldier at certain intervals (e.g., once per second).Further, the backpack can include a transmission system that transmitsthe location of the soldier, at the intervals, wirelessly to a centralprocessing system. What is actually received by the central processingsystem may not be completely accurate. For example, data packets may belost between the transmission system and the central processing systemand therefore complete information is not provided to the centralprocessing system.

What is actually received by the central processing system (e.g., whatposition information is actually received) can be compared with the datarecorder. Based on this comparison an accuracy level of the transmissionsystem can be computed. The set of raw technical performance data 120can be what the central processing system actually receives while theset of actual operation effectiveness assessment data 130 can be theaccuracy level. A correlation can be made that when information is sentfrom the backpack (e.g., the actual tested backpack, a backpack of thattype, etc.) the information will be delivered with the accuracy level.This correlation can be used to train the adaptive learning system 110.

In addition to training the adaptive learning system 110, the trainingcan be validated. In one example, the set of raw technical performancedata can be divided into halves—a training half and a validation half.The training half can be used to train the adaptive learning system 110and the validation half can be used to validate the adaptive learningsystem 110 after being trained. This can be done since it can be knownhow the adaptive learning system 110 should respond since the set ofactual operational effectiveness assessment data 130 is available.

In one embodiment, the adaptive learning system 110 is a deep learningsystem of at least five layers. The layers of the deep learning systemcan be distinct or subsequent layers can be combined previous layers. Anexample deep learning system can be a Convolutional Neural Network or aDeep Belief Network.

In one embodiment, once the adaptive learning system 110 is deployed theadaptive learning system 110 can continue to be trained and/orvalidated. This training of the adaptive learning system 110 can occuronline (e.g., while the adaptive learning system 110 is functioning) oroffline (e.g., when the adaptive learning system 110 is non-functioninguntil a session of subsequent training is finished). Additionally, howsuccessfully the adaptive learning system 110 functions can be used toinitially or subsequently train other adaptive learning systems.

FIG. 2 illustrates one embodiment of an environment 200 comprising theadaptive learning system 110. While the environment 100 of FIG. 1 can beconsidered as training the adaptive learning system 110 the environment200 can be considered how the adaptive learning system 110 functionsafter being initially trained and subsequently deployed.

Returning to the above example with GPS, a soldier can be deployed withthe backpack that transmits position information. This transmittedposition information can function as the set of raw technicalperformance data 210. While access to the transmitted positionalinformation can be available, access to the data recorder may not bepossible in a reasonable timeframe. Therefore, the adaptive learningsystem 110 can make a prediction on accuracy of the transmitted positioninformation. This prediction can function as the set of predictedoperational effectiveness assessment data 220. As shown in this example,the set of training technical performance data (e.g., the set of rawtechnical performance data 120 of FIG. 1) can be of a firstclassification (e.g., transmitted data) and of a second classification(e.g., data recorder data) while the set of raw technical performancedata 210 can be of the first classification and not of the secondclassification.

While relatively simple concepts are discussed above, it is to beappreciated by one of ordinary skill in the art that far more complexpredictions can be performed by a highly robust adaptive learning system110. Different sets and types of data can populate the set of rawtechnical data 210, such as data from different hardware elements,different systems, etc. Similarly, the set of predicted operationaleffectiveness assessment data 220 can be more robust with various levelsof predictions, types of predictions, confidences in predictions, etc.

FIG. 3 illustrates one embodiment of an environment 300 comprising theadaptive learning system 110. The environment 300 shows instrumenteddata from a testing and evaluation (T&E) event 310 that can beconsidered raw instrumented data that is provided to the adaptivelearning system 110. In addition, operational assessment data 320 fromthe T&E event (e.g., provided by human evaluators/event assessors) canbe inputted into the adaptive learning system 110. The operationalassessment data 320 can be available for a subset (less than a full set)of a test's duration (e.g., partial time). The output from the adaptivelearning system 110 can be a system of systems (SoS) characterization330 for a full set of the test's duration and/or for a remainingduration. The instrumented data 310 collection and characterization 330can occur from full event durations while the assessment data 320collection can take place at a fraction of event duration. Use of theadaptive learning system 110 can greatly reduce event evaluation costs,by eliminating human evaluators for the entire event duration. Humanevaluators can be used for an initial phase that provides training andvalidating data for the adaptive learning system.

The adaptive learning system 110, such as a deep learning architecture,can learn complex relationships between multi-dimensional spaces ofvarious functions including unsupervised learning and proactive (e.g.,automated) classification capabilities of highly complex data sets. Thedeep learning architecture and algorithms can be used to determineoperational performance of applications and services as well as systemof system network formation. This architecture can be trained at varioustimes including continuous time training (e.g., prepared for traininganytime) or infrequent time training.

In some scenarios it may be beneficial to have a repeatable, automatedcapability to evaluate emerging technologies with specific emphasis on aparticular sector of interest (e.g., auto manufacturing technologies foran automotive company). One manner to evaluate these technologies is todevelop measures of performance that are highly creatable withoperational performance. These measures of performance can beinterpolated to develop system threshold and objective technicalperformance specifications. Development of these measures of performancecan require an individual with both sound judgment and significantscientific knowledge. When individuals such as this are in short supplya system can be used to develop these measures of performance. Thissystem, however, may prove to be very costly and testing can be complex.

Aspects disclosed herein can be practiced for quick analysis ofoperational performance of complex systems and/or systems of systemsbased on a limited quantity of operational effectiveness evaluationdata. Operational performance can be enabled by complex correlation ofunderlying human and system functional activities. Pattern understandingcan be used in automated data classification and analysis through use ofthe adaptive learning system 110. The adaptive learning system 110, suchas a deep learning system, can be used toward automated operationaleffectiveness assessment of systems and the like.

Employment of the adaptive learning system 110 can result in the abilityto quickly learn complex relationships between operation effectivenessof systems (e.g., system of systems) and their various technicalperformance factors. This can also result in the ability to scaleprocessing time linearly with growth in experimentation data. Inaddition, this can result in the ability to exploit similarly-structuredmeasures of performance across various systems and/or applications whichallows for exploitation of synergies between different operational taskassessments. Additionally, this can also result in an ability to operateagainst level two raw data, commonly available by common instrumentationcapabilities.

FIG. 4 illustrates one embodiment of a configuration 400 comprising anumber of systems 410-430 and a parent system 440. The number of systems410-430 and/or the parent system can be individual adaptive learningsystems and/or function as a single learning system. In one embodimentthe configuration 400 represents a system of systems configuration,where the systems 410-430 are of the parent system 440 (e.g., are partof the parent system 440 (e.g., are subservices of a complex service),feed data to the parent system 440, etc.) The systems 410-430 can beconsidered information pipelines while the parent system 440 cancomprise or be part of the adaptive learning system 110 of FIG. 1. Thesystems 410-430 can be subordinate to the parent system 140 and/orsupply information to the parent system 440. While shown as separate,the parent system 440 can comprise the systems 410-430. In oneembodiment, the set of raw technical performance data 210 of FIG. 2 canbe a first set of raw instrumented data that is associated with a firstsubsystem (e.g., the 1^(st) system 410 that is part of the parent system440). A second subsystem (e.g., the 2^(nd) system 420 that is part ofthe parent system 440), that can be configured to not be identical tothe first subsystem, can be associated with a second set of rawinstrumented data that is also part of the set of raw technicalperformance data 210 of FIG. 2. The adaptive learning system 110 of FIG.1 can be configured to use the first set of raw instrumented data toproduce a first operational score and to use the second set of rawinstrumented data to produce a second operational score. Additionally,the adaptive learning system 110 of FIG. 1 can be configured to use thefirst operational score and the second operational score to make theeffectiveness prediction.

FIG. 5 illustrates one embodiment of an environment 500. Two phases orstages can be defined for the adaptive learning system 110 of FIG. 1 andas such the environment 500 can represent the adaptive learning system110 of FIG. 1 along with the effectiveness data that is functioning asthe overall operational effectiveness prediction. The first phase canpredict operational scores for individual services/subsystems and/orprovide an intermediate description. These individual services can beimplemented as a number of parallel pipelines (1 to N) that each uses adeep learning system (e.g., their own deep learning system). The secondphase can take the output from the individual services (e.g., theoperational scores and/or the intermediate description). And predictoverall operational effectiveness. Phase 1 and phase 2 can be part ofthe adaptive learning system 110 of FIG. 1.

In one embodiment, the adaptive learning system 110 of FIG. 1 can refinelearning behaviors and/or algorithms independently for phase 1 and phase2. In addition, the adaptive learning system 110 of FIG. 1 can identifyand isolate a weak performing pipeline (e.g., a pipeline that is notproviding reliable training). The impact of training from this weakperforming pipeline can be mitigated (e.g., not used in furthertraining, not used in further training until correct, etc.).

FIG. 6 illustrates one embodiment of an environment 600. The environment600 can be used to validate learning of the adaptive learning system 110of FIG. 1, such as when implemented as a deep learning architecture. Thedeep learning architecture can receive raw platform positioncoordinates/position reports technical data and a first set of accuracydata (e.g., brute-force computed accuracy of platform position given theposition and reports) as an input. After training, the deep learningarchitecture can output a second set of predicted accuracy data givenplatform locations and platform location reports. The sets of accuracydata can be used to cross-validate such that perceived and actualplatform location accuracies (e.g., one brute-force computed and onepredicted) are compared to one another. In this way, using the adaptivelearning system 110 of FIG. 1, one can predict system accuracy fromplatform location data and data reflecting function of communicationsystem performance absent further complex brute force computation ofaccuracy.

FIG. 7 illustrates one embodiment of a system 700 comprising a processor710 and a non-transitory computer-readable medium 720. In one embodimentthe non-transitory computer-readable medium 720 is communicativelycoupled to the processor 710 and stores a command set executable by theprocessor 710 to facilitate operation of at least one component that canperform an aspect disclosed herein (e.g., a validation componentconfigured to validate the adaptive learning system 110 of FIG. 1). Inone embodiment, at least one component that can perform an aspectdisclosed herein (e.g., a trainer component configured to train theadaptive learning system 110 of FIG. 1) can be implemented, at least inpart, by way of non-software, such as implemented as hardware by way ofthe system 700 or implemented as software retained on the non-transitorycomputer-readable medium 720. In one embodiment the non-transitorycomputer-readable medium 720 is configured to store processor-executableinstructions that when executed by the processor 710 cause the processor710 to perform a method disclosed herein (e.g., the method 800, themethod 900, and/or the method 1000 discussed below).

FIG. 8 illustrates one embodiment of a method 800 comprising threeactions 810-830. At 810 predicting a first operational score for a firstindividual service through use of a first service deep learning systemcan occur. At 820 predicting a second operational score for a secondindividual service through use of a second service deep learning systemthat is distinct from the first service deep learning system can occur.At 830 predicting an overall operational effectiveness though use of anoperational deep learning system can occur. The overall operationaleffectiveness can be based, at least in part, on the first operationalscore and the second operational score produced by their respectivelearning systems. In one embodiment, the first service deep learningsystem, the second service deep learning system, and the operationaldeep learning system are part of the same overall deep learning systemor are distinct learning systems (e.g., all three are distinct from oneanother).

In one embodiment, the first individual service and the secondindividual service can be on the same hierarchical level or on differenthierarchical levels. Examples of being on the same hierarchical levelcan include equivalent electronic devices that communicate with oneanother. Being on different hierarchical levels can include amaster-slave arrangement where a first device has dominion over a seconddevice.

FIG. 9 illustrates one embodiment of a method 900 comprising six actions910-960. A set of raw technical performance data and a set of actualoperational effectiveness data can be accessed at 910 and 920respectively. This accessing can be of data sent in response to arequest, data extracted from available sources, data located in a securelocation, etc. Similarly the accessing can include obtaining data,computing data, receiving data, etc.

At 930 training the adaptive learning system 110 of FIG. 1 (e.g.,functioning as a deep learning system that employs at least five layers)based on a relationship between at least part of the set of rawtechnical performance data and the set of actual operationaleffectiveness data can occur. At 940 validating the adaptive learningsystem 110 of FIG. 1 can occur after the adaptive learning system istrained at 930, and thereafter successful validation the adaptivelearning system 110 of FIG. 1 can be deployed at 950.

In one embodiment, accessing the technical data at 910 can comprisedividing the set of raw technical performance data into a first raw partand the second raw part. Training the adaptive learning system 110 ofFIG. 1 at 930 can be based on a relationship between the first raw partand the set of actual operational effectiveness data. Validating theadaptive learning system 110 of FIG. 1 at 940 can be testing theadaptive learning system 110 of FIG. 1 through observation of apredicted operation effectiveness by use of the second raw part. Afterbeing trained (e.g., after being trained and validated) the adaptivelearning system 110 of FIG. 1 can be deployed at 950. While deployedand/or after being deployed and then un-deployed the adaptive learningsystem 110 of FIG. 1 can be further trained at 930. At 960 collecting aset of actual technical performance data can take place as well asmaking an actual operation effectiveness prediction through use of theset of actual technical performance data by way of the adaptive learningsystem 110 of FIG. 1 while the adaptive learning system 110 of FIG. 1 isdeployed. In one example, the set of raw technical performance data(e.g., human produced data, such as from the backpack of the GPSexample) is from an initial duration of a test and evaluation event andthe set of actual technical performance data (e.g., produced frommathematical calculation, derived, at least in part, from the set of rawtechnical performance data, etc.) is from a remaining duration of thetest and evaluation event.

FIG. 10 illustrates one embodiment of a method 1000 comprising twoactions 1010 and 1020. At 1010 there is obtaining the set of rawtechnical performance data 210 of FIG. 2 while at 1020 there ispredicting a set of predicted operational assessment data (e.g., set ofpredicted operational effectiveness assessment data 220 of FIG. 2)based, at least in part, on the set of raw technical performance data210 of FIG. 2. This prediction can be made by way of a learning system(e.g., the adaptive learning system 110 of FIG. 2). The adaptivelearning system 110 of FIG. 1 can be trained from the set of rawtechnical performance data 120 of FIG. 1 (e.g., set of trainingtechnical performance data) with a set of training operationalassessment data (e.g., the set of actual operational effectivenessassessment data 130 of FIG. 1) and be a deep learning system.

In one embodiment, the set of training technical performance data istechnical performance data from an initial duration of a test andevaluation event while the set of raw technical performance data istechnical performance data from a remaining duration of the test andevaluation event. In one embodiment, the set of training operationalassessment data is human produced and/or the set of training operationalassessment data is produced from mathematical calculation. Using the GPSexample above, the set of training operational assessment data (e.g.,accuracy level) can be calculated through comparison of the datarecorder against what is transmitted and received wirelessly.

What is claimed is:
 1. A computer-implemented method, comprising:accessing a set of raw technical performance data; accessing a set ofactual operational effectiveness data; training an adaptive learningsystem based on a relationship between at least part of the set of rawtechnical performance data and the set of actual operationaleffectiveness data; deploying the adaptive learning system once trained;further training the adaptive learning system while the adaptivelearning system is deployed; collecting a set of actual technicalperformance data; and making an actual operation effectivenessprediction through use of the set of actual technical performance databy way of the adaptive learning system while the adaptive learningsystem is deployed, where the adaptive learning system is a deeplearning system that employs at least five layers, where the set of rawtechnical performance data is from an initial duration of a test andevaluation event, where the set of actual technical performance data isfrom a remaining duration of the test and evaluation event, where theadaptive learning system, when online, is configured to predict a set ofpredicted operational assessment data based, at least in part, on theset of raw technical performance data and where the adaptive learningsystem trained while the adaptive learning system is online.
 2. Thecomputer-implemented method of claim 1, comprising: validating theadaptive learning system after the adaptive learning system is trained;and deploying the adaptive learning system after validation of theadaptive learning system.
 3. The computer-implemented method of claim 2,comprising: dividing the set of raw technical performance data into afirst raw part and a second raw part, where training the adaptivelearning system based on the relationship between at least part of theset of raw technical performance data and the set of actual operationaleffectiveness data comprises training the adaptive learning system basedon a relationship between the first raw part and the set of actualoperational effectiveness data and where validating the adaptivelearning system comprises testing the adaptive learning system throughobservation of a predicted operation effectiveness by use of the secondraw part.
 4. The computer-implemented method of claim 1, where the setof raw technical performance data is human produced.
 5. Thecomputer-implemented method of claim 4, where the set of actualoperational effectiveness data is produced from mathematicalcalculation.
 6. The computer-implemented method of claim 1, where theset of actual operational effectiveness data is derived, at least inpart, from the set of raw technical performance data.